Electronic services such as, for example, web services, are used to provide clients with an increasing quantity and variety of commercial services. Such commercial services may include, for example, news, sports, financial information, entertainment information, photographs, videos, music, maps, legal information, employment information, and information about buying or selling various goods and services. Generally, electronic service providers operate by charging clients a fee for the services which they provide. When a client subscribes to a particular electronic service, the agreement between the service provider and the client is generally described in a service contract. The portion of such a contract that relates to the expected quality of the service is typically described in a document that is referred to as a service level agreement. Representing service level agreements electronically is a recent development driven by the increasing popularity of electronic services.
The service level agreement may include a number of different metrics, which are categories in which the service provider's performance is to be measured. By way of example and not limitation, such metrics may include response time, response size, and response quantity. Another metric may be, for example, availability, which measures how often the service is available to clients. Other metrics may measure the quality of responses. For example, if a service is expected to return images to its clients, the quality of the images may be measured based on factors such as, color quality, depth, accuracy, precision, and pixel counts. For each metric, the service level agreement may include a standard to which the service provider is expected to conform. For example, for a response time metric, the service level agreement may require that the service provider generate responses within 10 seconds of receiving a request from a client.
Each metric may be measured and computed in accordance with a number of individual measurement points herein referred to as probes, which are values that are determined based on interactions between the service provider and its clients. For example, the response time metric may be calculated in accordance with an incoming probe and an outgoing probe. The incoming probe may correspond to the time at which a service request from a client is received at the service provider. The outgoing probe may correspond to the time at which a response to the service request is returned from the service provider to the requesting client. The response time metric may then be calculated by subtracting the incoming probe value from the outgoing probe value.
In addition to measuring the service provider's performance, probes may also be used to ensure that clients are submitting valid requests to the service provider. For example, if a service provider has entered into an agreement to provide maps of Europe to a client, then the service provider should not be penalized for failing to provide a response when the client requests a map of the United States. Thus, for example, a map metric may have a corresponding incoming probe which confirms that the client has requested a valid latitude and longitude within the area that the service provider has agreed to. Probes that measure the service provider's responses may be referred to as provider response probes, while probes that confirm that the client is requesting valid data may be referred to as client validation probes.
An exemplary prior art system for monitoring the performance of a service provider is shown in FIG. 1. As shown, a client 103 has entered into a service agreement with service provider 101. Monitoring component 105 uses service level agreement 104 to monitor the performance of service provider 101 to ensure that service provider 101 is conforming to the agreed upon performance standards. Monitoring component 105 may be operated by client 103, service provider 101, or by a third party that is trusted to evaluate the performance of the service provider 101. To enable evaluation of its performance, service provider 101 collects and maintains monitoring data 102. Monitoring data 102 is data that allows service provider 101 to be evaluated according to the metrics set forth in service level agreement 104.
Client 103 may avail itself of the services provided by service provider 101 by submitting a service request 108 to service provider 101. Service request 108 may be, for example, a simple object access protocol (SOAP) packet. Service provider 101 may receive, evaluate, and process request 108, and then eventually return a service response 109 back to client 103. For example, client 103 may submit a service request 108 for a map of the United States, and service provider 101 may then generate and return the map of the United States to client 103 as service response 109. Monitoring component 105 monitors the communications between client 103 and service provider 101 to identify any service requests 108.
When monitoring component 105 identifies a service request 108, monitoring component 105 examines service level agreement 104 to identify any metrics that are implicated by the service request 108. For example, if service request 108 is for a map of the United States, then service request 108 may implicate a map metric that measures map detail, map dimensions, and map size and also confirms that a valid latitude and longitude have been requested by the client 103. Such a service request 108 may also implicate, for example, a service response time metric. Monitoring component 105 may then query monitoring data 102 to obtain the data necessary to evaluate service provider 101 based on the implicated metrics. Generally, instructions for gathering monitoring data 102 and evaluating the metrics are set forth and organized in a service level agreement language.
A limitation of conventional service level agreement languages is that they do not provide sufficient description of metrics to enable monitoring component 105 to compile its own monitoring data from the contents of the service requests 108 and the service responses 109. Rather, conventional service level agreements languages force monitoring component 105 to collect pre-compiled monitoring data 102 from service provider 101 or to collect only prescriptive monitoring data that cannot vary between individual service level agreements. For example, to evaluate the response time metric described above, conventional service level agreement languages may instruct monitoring component 105 to retrieve the incoming probe value and the outgoing probe value from monitoring data 102 at service provider 101. The conventional service level agreement languages will generally only provide enough information to enable monitoring component 105 to identify and collect the probes from service provider 101, without having any understanding of what the probe values represent or how they were determined.
Relying on service provider 101 to compile its own monitoring data 102 creates a number of problems. Obviously, there is a danger that service provider 101 could conceivably tamper with monitoring data 102 to produce results which are more favorable to service provider 101. For example, to decrease its response time metric, service provider 101 could conceivably subtract a number of seconds from the outgoing probe value that it provides to monitoring component 105. To reduce the risk that service provider 101 will employ such tactics, monitoring component 105 or another entity may “audit” service provider 101 to inspect the processes by which it compiles its monitoring data 102 and attempt to verify that such processes are, in fact, legitimate. However, auditing service provider 101 may often prove to be an expensive and time consuming proposition. Moreover, even if service provider 101 is audited and its data compilation procedures are found to be legitimate, there is a danger that service provider 101 will simply change its data compilation procedures after the audit is performed. This danger creates the need for repeated auditing of service provider 101, which further drives up the time and expense of the monitoring process. In addition to these dangers of data tampering, forcing the service provider 101 to generate its own monitoring data increases the size of code that must be generated for service provider 101 and the number of tasks that must be performed by service provider 101, thereby increasing operating expenses and diverting resources from other aspects of the service provider's operations.